Learning Expected Emphatic Traces for Deep RL

Off-policy sampling and experience replay are key for improving sample efficiency scaling model-free temporal difference learning methods. When combined with function approximation, such as neural networks, this combination is known the deadly triad potentially unstable. Recently, it has been shown that stability good performance at scale can be achieved by combining emphatic weightings multi-step updates. This approach, however, generally limited to complete trajectories in order, compute required weighting. In paper we investigate how combine non-sequential, off-line data sampled from a buffer. We develop weighting replay, time-reversed n-step TD algorithm learn show these state reduce variance compared prior approaches, while providing convergence guarantees. tested approach on Atari 2600 video games, observed new X-ETD(n) agent improved over baseline agents, highlighting both scalability broad applicability of our approach.